Skip to content

new vg combine, adding new options - #4965

Open
xinzilan wants to merge 7 commits into
masterfrom
new_vg_combine
Open

xinzilan wants to merge 7 commits into
masterfrom
new_vg_combine

Conversation

@xinzilan

@xinzilan xinzilan commented Jul 14, 2026 •

Copy link
Copy Markdown

Changelog Entry

To be copied to the draft changelog by merger:

  • vg combine gained options for stitching chunked assemblies and graphs: -f/--connect-fragments (now the default mode), -s/--shared-nodes, -m/--merge, and -u/--fusion.
  • vg concat is removed (use updated vg combine instead)

Description

This PR adds four new options to vg combine to support our chunked-assembly stitching workflow.

  • -f/--connect-fragments (default mode): Treats paths that share the same sample/locus/haplotype/phase block but differ only by subrange — e.g. CHM13#0#chr22 and CHM13#0#chr22[17475777] — as fragments of one path. It sorts the fragments by start offset, concatenates them in order, adds the missing inter-fragment edges, and rewrites the path name to a single [start-end] subrange covering the union. This runs by default whenever none of -c/-p/-m is given.

  • -s/--shared-nodes: Modifies the default (-f) mode so node IDs are not renumbered when merging. Node IDs appearing in both inputs must carry the same sequence; one copy is kept. This is intended for vg chunk output that shares boundary nodes across chunks, where the graph (rather than the assembly) was chunked. Fragments are merged into one path, trimming duplicated boundary steps. Valid only with the default mode; cannot be combined with -c/-p/-m.

  • -m/--merge: Overlap-aware stitching for chunked-assembly graphs that share REFERENCE-sense paths (e.g. CHM13#0#chr22). Inputs are bucketed by REFERENCE identity (sample/locus/haplotype), so chunks of different chromosomes (e.g. CHM13#0#chr21 and CHM13#0#chr22) can be passed together — each reference is stitched independently and kept as a separate path in the output. Within a reference, it sorts inputs by REFERENCE start offset, validates that each chunk boundary is a single chain node shared by every path (and equal to the REFERENCE end), trims each right chunk's leading node by the REFERENCE-offset overlap, connects the left end to the trimmed right start, then merges fragments like -f.

  • -u/--fusion (requires -m): Instead of adding an edge between the left end node and the right chunk's trimmed start node, fuses them into a single node carrying the concatenated sequence.

@faithokamoto

Copy link
Copy Markdown
Contributor

Are we ready to remove vg concat, then? #4932 (comment)

@adamnovak

adamnovak commented Jul 17, 2026 •

Copy link
Copy Markdown
Member

Yeah I think this will replace vg concat.

@adamnovak adamnovak left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I found a few problems with this.

  1. There aren't any tests. There need to be tests for different edge cases of path handling (paths that overlap between input graphs, paths with gaps between them, paths that use phase blocks and don't really want to be connected, paths that are in pieces in their original input graphs) so that we can ensure that the code actually handles paths in the way we expect and produces the appropriate errors when graphs shouldn't be able to be combined.
  2. Because there aren't any tests, I think the code is able to do the wrong things with some of those situations. I think it's able to produce paths where their subranges don't actually reflect the total length of all their visits, for example. It's also claiming to be able to change the sense of paths from HAPLOTYPE to REFERENCE.
  3. The comments often substitute a summary of the code for an explanation of the abstraction that the code is supposed to provide to its caller to work with. This includes the user-facing documentation for the options.
  4. In general the different modes don't share code where it seems like they should, and don't explain why they can't. It looks like -f mode was implemented to combine graphs and join up paths, and then -m mode was implemented to combine fragments along a reference-sense backbone and join up paths, and besides the path-joining function they share basically nothing, with similar problems solved by main force with completely different code. I would expect to see one main implementation of loading and combining the graphs, which is able to do different things at different points by e.g. checking a condition and deciding whether or not to enforce constraints around a single REFERENCE set of paths. I'm not sure the reference mode can't just be a pass, rather than its own nearly-complete implementation.
  5. We're approaching the level of complexity where we no longer want all the code to live in the subcommand file. There probably should be a widget class in src or some new algorithms in src/algorithms to hold the guts of this.

Comment thread src/subcommand/combine_main.cpp Resolved
Comment thread src/subcommand/combine_main.cpp Resolved
Comment thread src/subcommand/combine_main.cpp Resolved
Comment thread src/subcommand/combine_main.cpp Resolved
Comment thread src/subcommand/combine_main.cpp Resolved
Comment thread src/subcommand/combine_main.cpp Resolved
Comment thread src/subcommand/combine_main.cpp Resolved
Comment thread src/subcommand/combine_main.cpp Resolved
Comment thread src/subcommand/combine_main.cpp Resolved
Comment thread src/subcommand/combine_main.cpp Resolved
@faithokamoto

Copy link
Copy Markdown
Contributor

I've taken care of removing vg concat, so that now the two changes will merge at the same time. Shouldn't affect any of the vg combine updates.

@faithokamoto

Copy link
Copy Markdown
Contributor

I did a quick check through the old issues, and this may be relevant to #2688?

@adamnovak

Copy link
Copy Markdown
Member

Yeah, this should fix #2688.

@faithokamoto

Copy link
Copy Markdown
Contributor

Ah, this line is going to need to be replaced by vg combine. I defer the decision of how to do that to the people who understand what is happening here.

vg concat x.vg y.vg z.vg >q.vg

@xinzilan
xinzilan requested review from adamnovak and removed request for adamnovak July 18, 2026 01:32
@faithokamoto

Copy link
Copy Markdown
Contributor

#2677 may also be relevant, if this PR means we could do the workflow Adam said was impossible here: #2677 (comment)

@xinzilan
xinzilan requested a review from adamnovak September 16, 2026 11:30

@adamnovak adamnovak left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this will work, and is possible to understand.

I also think the core abstractions could stand to be redesigned and, especially, more clearly explained. What's an accumulator? What's the general idea of the approach that a person will need to know to understand what the methods are for? Can we avoid weird states like an un-described Chunk?

The tests are thorough, but grouped oddly, and they are probably too thorough in easy areas while missing some cases that are actually important. We have 105 new command-line tests, many of which seem to be testing the same basic questions as unit tests. Tests are there for a lot of the error message strings, but verify the wording of the string rather than the presence of the actual path name/node number/etc. There are unit tests for input validation functions but only for valid and not invalid inputs.

I only gave the core implementation code a cursory scan.

I'm putting this in as a comment because I don't know if it makes sense for you to actually try and do all this cleanup @xinzilan. We want to merge this and it seems like it will, plausibly, work, and the structure isn't so terrible that it couldn't be maintained. I think you'd learn a lot form making all the changes I recommend, but most of the issues are the sort of issues that come out of a code generator. You probably could set the code generator to fixing them and get rid of most of them, but what I really want to convey is that it would be useful if you learned to spot these sorts of things and shake them out of code before submission.

If it were me, I would re-comment the headers, cut almost all the CLI tests, maybe make a few of the refactoring changes if it's clear that those give a more coherent final product, and maybe leave the rest alone? If you want to do a really good job, you can define some custom exception types for all the error messages, move some of the hundred CLI tests about error messages over into C++ unit tests about exception messages instead, and keep just the CLI tests for things about the CLI that the unit tests can't get at usefully.

But if @benedictpaten thinks we need the graph merging as soon as possible (which we do), we can also just merge this and postpone that sort of cleanup until later/never.

Comment thread src/algorithms/fuse_nodes.hpp Resolved
Comment thread src/subcommand/combine_main.cpp Outdated
}
logger.error() << "unrecognized --seam policy \"" << value
<< "\"; expected one of: renumber, shared, trim." << endl;
return SeamPolicy::RENUMBER; // not reached; logger.error() exits

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be nice if the comment didn't have to contradict the code like this.

We don't have std::unreachable() yet.

We should mark the destructor for the error stream [[noreturn]], but I don't think the compiler is guaranteed to be able to figure out that that means subsequent statements are unreachable. Even if it could, right now we're using one type with a flag for whether to exit, so we'd have to refactor it to be a special type for fatal errors. (See

vg/src/log.hpp

Lines 60 to 64 in d08602f

~cerrWrapper() {
if (exit_on_destruct) {
exit(EXIT_FAILURE);
}
}
)

I would do something like:

Suggested change
return SeamPolicy::RENUMBER; // not reached; logger.error() exits
throw std::logic_error("unreachable"); // logger.error() stream exits after logging

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done: I replaced the return with the throw std::logic_error("unreachable") you suggested.

Comment thread src/unittest/combine.cpp Outdated
return chunk;
}

/// Create a path with full metadata and walk it over the given handles.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Walking the path would mean following it after it was created, not creating it so that it goes a certain way. You could maybe say "run it over the given handles", but that's also kind of confusing.

I would try:

Suggested change
/// Create a path with full metadata and walk it over the given handles.
/// Create a path with the given metadata, that visits the given steps.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done: the comment now says it creates a path that visits the given steps.

Comment thread src/unittest/combine.cpp Outdated
return path;
}

/// Create a REFERENCE-sense path, the sense --seam trim positions a chunk by.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is part of the unit test, so it shouldn't really be cross-referencing the command line flags for vg combine. There's an abstraction barrier between the unit tests and the CLI.

Suggested change
/// Create a REFERENCE-sense path, the sense --seam trim positions a chunk by.
/// Create a REFERENCE-sense path.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done: the comment no longer mentions the --seam trim flag.

Comment thread src/unittest/combine.cpp Outdated
return names;
}

TEST_CASE("compute_path_length adds up what a path visits", "[combine]") {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But nodes themselves can't be summed.

Suggested change
TEST_CASE("compute_path_length adds up what a path visits", "[combine]") {
TEST_CASE("compute_path_length adds up visited node lengths", "[combine]") {

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done: I used your wording, and the function moved to libhandlegraph as you suggested.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done: I used your wording, and the function moved to libhandlegraph as you suggested in the following comment.

Comment thread test/t/51_vg_combine.t Outdated
vg convert -p $C/fb.gfa > $C/fb.pg 2>/dev/null
vg combine $C/whole.vg $C/fa.pg $C/fb.pg 2>$C/collide.err >/dev/null
is "$?" "1" "a merged name that is already taken is rejected"
is "$(grep -c 'already an unrelated path' $C/collide.err)" "1" "the error names the path standing in the way"

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This isn't actually checking that the path name appears in the message.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This error can't happen anymore as circularity is out of the key: a path with the merged name always has the same key as the fragments, so it is one of them. I replaced the error with an internal assertion and removed the test.

Comment thread test/t/51_vg_combine.t Outdated
vg construct -r small/x.fa -v small/x.vcf.gz > x.vg
vg construct -r small/x.fa -v small/x.vcf.gz | vg paths -d -v - > y.vg
vg construct -r small/x.fa -v small/x.vcf.gz | vg paths -d -v - > z.vg
C=combine

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We probably should explain why we would store a one-word value in a one-letter variable. It looks like we're trying to keep our test scratch work in this directory?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done: yes, it's the scratch directory for the test's files. I renamed it to SCRATCH and added a comment saying so.

Comment thread test/t/51_vg_combine.t Outdated
Comment on lines +236 to +237
printf "$RS"'S\t5\tGGTT\nS\t6\tA\nS\t7\tC\nS\t8\tGGGGG\nL\t5\t+\t6\t+\t0M\nL\t5\t+\t7\t+\t0M\nL\t6\t+\t8\t+\t0M\nL\t7\t+\t8\t+\t0M\nP\tGRCh38#0#chr1[8-18]\t5+,6+,8+\t*\nP\tHG002#1#chr1#0[8-18]\t5+,7+,8+\t*\nP\tHG002#2#chr1#0[12-18]\t6+,8+\t*\n' > $C/trb.gfa
printf "$RS"'S\t1\tAAAA\nS\t2\tC\nS\t3\tG\nS\t4\tGGGGG\nL\t1\t+\t2\t+\t0M\nL\t1\t+\t3\t+\t0M\nL\t2\t+\t4\t+\t0M\nL\t3\t+\t4\t+\t0M\nP\tGRCh38#0#chr1[0-10]\t1+,2+,4+\t*\nP\tHG002#1#chr1#0[0-10]\t1+,2+,4+\t*\nP\tHG002#2#chr1#0[0-5]\t1+,3+\t*\n' > $C/tlb.gfa

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we have to inline GFA in the tests, I would replace these with heredocs that someone can edit and read.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done: the inline GFA is all in heredocs now, one line per record. Fields are separated by spaces there, and tr turns them into tabs, so editors can't mix up tabs and spaces.

Comment thread test/t/51_vg_combine.t Outdated
Comment on lines +285 to +287
vg combine -s trim -u $C/bl_dangle.pg $C/br.pg >/dev/null 2>$C/fuse.err
is "$?" "1" "--fuse rejects a boundary node with an edge on the welded side"
is "$(grep -c 'weld would have to drop' $C/fuse.err)" "1" "the error says what the weld would drop"

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are some of these tests overlapping with the unit tests? We can test everything as a unit test and again as a command-line test, but we're adding 105 command line tests here, which seems like enough that we might have to start to think about total runtime, and it might be best to focus on the things that we think still could possibly be wrong after the unit tests pass. If there was a good way to turn some of these into unit tests instead, and we could find a way to verify error messages from Catch tests (maybe by throwing exceptions instead of using the logger and then logging from a catch block?), things might be more manageable.

Maybe it is in fact better to constrain all important aspects of the program's behavior with tests? But if we want tests at that scale we might need to adopt something other than Bash-TAP to manage them, because Bash-TAP doesn't let us do things like run a particular failing test out of our 115 distinct tests.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done: the errors are exceptions now, so I moved the error cases into unit tests and dropped the command line tests that only re-checked what the unit tests cover. The 45 that are left check what the unit tests can't: option parsing, each mode end to end, an error from the combining code reaching the command line, and graphs made by other vg subcommands.

Comment thread test/t/51_vg_combine.t Outdated
Comment on lines +368 to +373
# Whatever VPKG can load, combine can merge.
vg convert -p $C/adj1.vg > $C/adj1.pg
vg convert -a $C/adj2.vg > $C/adj2.hg
vg combine $C/adj1.pg $C/adj2.hg > $C/o_fmt.vg 2>/dev/null
is "$(vg paths -Ev $C/o_fmt.vg)" "$(vg paths -Ev $C/o_real.vg)" "PackedGraph and HashGraph inputs merge the same way"
is "$(vg stats -N $C/o_fmt.vg)" "32" "and keep the same nodes"

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we really want to test that, we need to try some of the non-mutable graphs (.xg, .gbz).

If we're just going to test that it works when the graph is a HashGraph and when it is a PackedGraph, I don't think it's worth testing at all.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done: I removed it. Combining needs mutable graphs, so read-only formats like GBZ are rejected, and there's already a test for that.

@adamnovak adamnovak left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this will work, and is possible to understand.

I also think the core abstractions could stand to be redesigned and, especially, more clearly explained. What's an accumulator? What's the general idea of the approach that a person will need to know to understand what the methods are for? Can we avoid weird states like an un-described Chunk?

The tests are thorough, but grouped oddly, and they are probably too thorough in easy areas while missing some cases that are actually important. We have 105 new command-line tests, many of which seem to be testing the same basic questions as unit tests. Tests are there for a lot of the error message strings, but verify the wording of the string rather than the presence of the actual path name/node number/etc. There are unit tests for input validation functions but only for valid and not invalid inputs.

I only gave the core implementation code a cursory scan.

I'm putting this in as a comment because I don't know if it makes sense for you to actually try and do all this cleanup @xinzilan. We want to merge this and it seems like it will, plausibly, work, and the structure isn't so terrible that it couldn't be maintained. I think you'd learn a lot form making all the changes I recommend, but most of the issues are the sort of issues that come out of a code generator. You probably could set the code generator to fixing them and get rid of most of them, but what I really want to convey is that it would be useful if you learned to spot these sorts of things and shake them out of code before submission.

If it were me, I would re-comment the headers, cut almost all the CLI tests, maybe make a few of the refactoring changes if it's clear that those give a more coherent final product, and maybe leave the rest alone? If you want to do a really good job, you can define some custom exception types for all the error messages, move some of the hundred CLI tests about error messages over into C++ unit tests about exception messages instead, and keep just the CLI tests for things about the CLI that the unit tests can't get at usefully.

But if @benedictpaten thinks we need the graph merging as soon as possible (which we do), we can also just merge this and postpone that sort of cleanup until later/never.

xinzilan and others added 3 commits October 10, 2026 00:43
# Conflicts:
#	autocomp.bash
#	autocomplete.sh
#	doc/vgmanmd.desc.md
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@xinzilan
xinzilan requested a review from adamnovak November 6, 2026 18:53

@adamnovak adamnovak left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't make it fully through this but I found some things to change.

I'm also attaching an attempt at an automated review of the full PR using #5103. I haven't read it yet, but we could use it to evaluate that PR or maybe mine it for ideas.

pr4965-review.md

return clean;
}

/// Fuse two nodes into one node.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We might not want to duplicate doc comments on definitions.

Comment on lines +143 to +145
} catch (const std::invalid_argument& e) {
logger.error() << e.what() << endl;
}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is meant to catch the not-enough-inputs signal from the combiner, but it's pretty general, so if something goes really wrong inside the combiner we might hit this instead of getting a stack trace. (Assuming our stack trace on exception stuff actually works...)

Maybe the CLI layer should be responsible for handling not having enough inputs (so it can print a useful message, and so we have the option to add some kind of override flag later in case we want to combine zero or more things in scripts), and the combiner should do something sensible with one input (pass it through) and zero inputs (produce an empty graph).

Comment thread src/unittest/combine.cpp
Comment on lines +4 to +10
///
/// GraphCombiner::combine() is covered end to end by test/t/51_vg_combine.t.
/// These tests drive the steps underneath it, whose intermediate results -- the ID
/// shifts, the trimmed boundary, the merged coordinates -- the command line cannot see.
///
/// Error branches are absent on purpose: logger.error() exits the process, which would
/// take the whole test binary with it. They live in test/t/51_vg_combine.t.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This isn't really that useful here. This is always the philosophy with the unit test files. If we have to reiterate it, we should do that more simply.

If we really want to test error logging we could adjust the logging system to let us capture errors during testing. Maybe we make it throw some kind of LoggedFatalError. Maybe we do that all the time and we catch and do the actual output somewheew in main()? But we don't need to do that to merge this.

Comment thread src/unittest/combine.cpp

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is more tests than I am going to read. There hopefully aren't a lot of problems lurking in here.

Comment thread src/combine.hpp
int64_t max_node_id = 0; ///< Largest node ID used in dest so far.
std::map<BasePathKey, SeamState> seams; ///< For each base path, the node, orientation, offset and input it ends at so far.

friend class unittest::TestGraphCombiner;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rather than forward-declaring and being friends with the test class that exposes all this stuff, can't we make it protected instead of private so the test class can expose it? If it constitutes a unit that can need testing, it probably isn't an integral enough part of the containing class's unit to be private.

Comment thread test/t/51_vg_combine.t
Comment on lines +18 to +47
tr ' ' '\t' > $SCRATCH/ml.gfa <<'EOF'
H VN:Z:1.1 RS:Z:GRCh38
S 1 AAAA
S 2 C
S 3 G
S 4 GGGGG
L 1 + 2 + 0M
L 1 + 3 + 0M
L 2 + 4 + 0M
L 3 + 4 + 0M
P GRCh38#0#chr1[0-10] 1+,2+,4+ *
P HG002#1#chr1#0[0-10] 1+,2+,4+ *
P HG002#2#chr1#0[0-10] 1+,3+,4+ *
P HG005#1#chr1#0[0-10] 1+,3+,4+ *
EOF
# Right chunk including only HG002
tr ' ' '\t' > $SCRATCH/mr.gfa <<'EOF'
H VN:Z:1.1 RS:Z:GRCh38
S 5 TTTT
S 6 A
S 7 C
S 8 GGGGG
L 5 + 6 + 0M
L 5 + 7 + 0M
L 6 + 8 + 0M
L 7 + 8 + 0M
P GRCh38#0#chr1[10-20] 5+,6+,8+ *
P HG002#1#chr1#0[10-20] 5+,7+,8+ *
P HG002#2#chr1#0[10-20] 5+,6+,8+ *
EOF

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These are legible, but is this really better than committing file fixtures?

Comment thread src/combine.hpp
bool is_circular = false; ///< Whether this base path is circular.
};

/// An input's reference path: its name, its base path, its first and last nodes, where it starts and ends on that base path, and whether it is circular.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can't list more than like 2 things in a class description like this; we need to explain what the class represents. We already have the list of fields in the body. I guess this is for holding metadata about a reference path as it exists within a single input graph?

Comment thread src/combine.hpp
Comment on lines +147 to +163
/// Throw InputConflictError if the input holds more than one fragment of a logical path.
void check_no_split_paths(const Input& input) const;
/// Throw PathLengthMismatchError if a path in the input has a subrange end other than its start plus its sequence length.
static void check_subrange_matches_length(const Input& input);
/// Describe `input`'s reference path. Throw ReferencePathError if it does not have
/// exactly one, or if that path visits its first or last node in reverse.
ReferenceDescription describe_reference(const Input& input) const;
/// Group each base path's chunks together and sort them by start position.
void sort_by_base_path_offset(std::vector<Chunk>& chunks);
/// Throw ChunkOverlapError if two neighbouring chunks of one base path spell different sequences where they overlap.
/// The chunks must already be sorted by sort_by_base_path_offset().
void check_overlaps_agree(const std::vector<Chunk>& chunks) const;

/// Throw ChunkOverlapError if any path in the chunk does not start at its reference path's first node.
void check_all_paths_start_at_reference_first_node(const Chunk& chunk) const;
/// Throw ChunkOverlapError if any path in the chunk does not end at its reference path's last node.
void check_all_paths_end_at_reference_last_node(const Chunk& chunk) const;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess each different check is in its own function so that we can then write unit tests that make sure the check catches the problems it is looking for?

That sort of makes sense, but then we get all these extremely specific check methods. And they're interleaved with other things.

Can't we just have one validation function? Or maybe one for inputs and one for chunks? Maybe they could be methods of inputs and of chunks? Or Input could have a validate() and Chunk could override it? We can still check all the cases and maybe make sure different kinds of problems have appropriate exception types.

Comment thread src/combine.cpp
Comment on lines +218 to +225
// Group by GroupKey, recording each path's start offset and name.
map<GroupKey, vector<pair<offset_t, string>>> groups;
input.graph->for_each_path_handle([&](const path_handle_t& path) {
groups[group_key_of(*input.graph, path)]
.emplace_back(path_start_offset(input.graph->get_subrange(path),
input.graph->get_phase_block(path)),
input.graph->get_path_name(path));
});

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isn't this duplicative with similar passes elsewhere? How much of the validation ends up re-doing work that the actual operation will need to do?

@faithokamoto

Copy link
Copy Markdown
Contributor

I skimmed Adam's reviews and, wait, we have a type called Input now? There's really nothing else we can name it? That's incredibly nonspecific.

@adamnovak

Copy link
Copy Markdown
Member

@faithokamoto I think it's fine because where it's defined it's inside the class that it's an input for. So you address it as GraphCombiner::Input everywhere outside GraphCpmbiner, and as Input when you are inside GraphCombiner.

It's still a bit of a non-idea abstraction ("What's a GraphCombiner::Input? Well it's just whatever a GraphCombiner needs as an Input!), but it's probably one of the least bad kinds.

@adamnovak

Copy link
Copy Markdown
Member

I looked over the auto-review findings, and of those I think we should:

  1. Add a unit test for an input containing an empty non-reference path (which the robot thinks will end up with us trying to get a step on the path).
  2. Figure out whether we really want node fusing to make non-reference paths that begin/end at one of the nodes being fused come out covering the new, longer node. I think we do, but we might want to note it somewhere. The alternative is figuring out if any paths don't continue into the next chunk and not fusing at those points.
  3. Add a unit test for haplotype paths of the same name in each chunk, in trim mode. Or say that isn't allowed. The robot thinks the computed subranges will come out wrong.
  4. Add a unit test for a one-node interior chunk, or say this isn't allowed.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Make vg combine handle subpaths and link chunked graphs (for chunked augmentation)

3 participants